Hosting public site, you’ve dealt with them already. Using your favorite search engine, you’re indirectly subject to their work as well. Crawlers are bots, querying and sometimes mapping your site, completing some search engine database. When I started looking at the subject, a few years back, you only had to know about /robots.txt, to potentially prevent … Continue reading Crawlers